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Basic Syntax 



S^rotsox: 

• Very C-like. 

func£ion( 'name', level, coptional Info ) = 'search terms and pattern' 
"wo valid search functions appid and fingerprint: 

• appidCchat/icq', 8.5, wireshark='icq f , chatproc='ICQ') = 

19 ] icq/c and $icq; 



• fm gerprint( 'finger prin t/p hone/nokia/gen eric 7.0) - 

’user-agent: nokia ' or 
| profile : 
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Namina Conventions 



XKS Appid ! s are named using a 
pseudo directory convention. 

/ applica tion = lype/sub = type/ name 
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Levels 



r 



Levels are 1.0 - 9.9 with lower numbers having a higher priority. This 
allows multiple signatures to match a piece of traffic and only the 
most specific appid will be applied. An example might be: 

9.9 Yahoo 

9.8 Yahoo/chat 

9.7 Yahoo/chat/incoming 



Since the Yahoo/chat/incoming has the lowest level, the traffic will be 
labeled as yahoo/chat/incoming 
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Basic Search Patterns 






XKEYSCORE supports Boolean operations and regular 
expressions 

Raw text must be encapsulated between single quotes 

• 'search term ' 

Terms can be combined with Boolean logic 



• 'search term ' and 'another term ' 

• 'search term' or 'another term' 



Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123 



Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123 

Binary and Reaex Pattern 












Binary patterns can be represented by putting a \x in front of each binary 
value 

• '\xff\xff\x00\x02' 

Note: Unlike C, no double back slashing required 
/regex/ 
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CHAINWORDs 1 




You can assign a pattern to a variable (CHAINWORD) and reuse 
the variable in many patterns. 

• $sip = 'via: sip' and cseq: ' and ' SIP/2 1 ; 

Now we can use this variable in future definitions: 

• appid( s voip/s!p 7.2 ) - $sip; 

• appidCvoip/sip/invite t 6.9) = $sip and ’INVITE 1 ; 
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Built in functions I 



ip( expr ) Matches against an IP Address looks in to address and 

from address in the session headere 

• iff '10. 10. 10.1' ); 

Matches against the Destination/To port. Note this 
must be a numeric representation of a port. 

• toport( 1920 ); 

Matches against the Source/From port. Note this must 
be a numeric representation of a port. 

° fromport( 80 ); 

Matches against the either port. Note this must be a 
numeric representation of a port. 

•port( 6667 ); 



next_protocol( expr ) 


Matches against the integer version of the next 
protocol. 

• next_protocol( 250 ); 


protocol ('text') 


Will only work for IP next protocol names as 
defined in the IANA next protocol numbers 
document 

• protocol('tcp'); 
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toportf expr ) 



fromport( expr ) 



port( expr } 
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Built in functions 




email_address(sel) 


permutes just like strong selector (just like 
DECODEQRDAIN 


mac_address(addr) 


Tasks a mac address 


smac(addr) 


dmac(addr) 


ip(addr) 


tasks this IP address (either to or from) 


from_ip(addr) 


tasks this IP address only when it is the originator 


to_ip(addr) 


tasks this IP address only when it is the destination 
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r r 



More built in functions 



first(expr) 


Matches against a pattern at the beginning of 
the session 


Ipos(expr) 


Matches against a pattern at the beginning of 
each line (\n) 


posf expr ) 


expression occurs at offset X in the session 

• pos(’Hello') == 5, 

• pos(/Good.*Grief/) <= 10 


between ( expr ) 


• between{'Hello', 'World', 10, 100) 

Separation between 'Hello' and 'World' is 
greater than or equal to 10 bytes and less 
than or equal to 100 bytes 

This is the same as using the following regular 
expression: 

• /Hello.{10,100}World/ 


'term'c 


Does a case sensitive match of the term 


'terrn'u 


Treats the term as UTF-16 
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Predefined Chainwords 



r 



There are a number of chainwords predefined for convenience: 



$tcp 


• $http_delete 


$udp 


• $http_trace 


$icmp 


• $http_head 


$sctp 


• $http_options 


$rpc 


• $ http partial 


$arp 


• $vbulletin 


$ssl 


• $mime._type 


$http_cmd 


• $user_agent 


$http 




$http_get 




$http_put 




$http„post 
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Example I 



appid('voip/sip/IMS', 6.0, wireshark='sip') = 

fvia: sip' or 'v: sip 1 ) and ’cseq:’ and ( 
'p-access-network-info:' or 
'p-cal led-party-icl : r or 
'p-charging-vector:' or 
'p-charging-vector-addresses:' or 
’p- media -authorization: 1 or 
'security-verify:' or 
'security-server:' or 
'security-client:' or 
'service-route: 1 or 
'record -route:' and 'pcscf' or 
'record -route:' and 'scscf' or 
'contact:' and 'pcscf or 
'contact:' and 'scscf' or 
'proxy-authorization:' and 'pcscf or 
'proxy-authorization:' and 'scscf' or 
'path:' and 'pcscf' or 
'path:' and 'scscf' 
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Example 




a ppidC'voi p/skinny/keep-alive', 3.0, wireshark= skinny ) = 

toport(2000 : and 

first( , \x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 l ) 



a ppidC'voi p/ski nny/keep-a I ive-ack : , 3.0, wireshark=’skinny ) = 

fro mport( 2000) and 

first( , \x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00’) 



appid(’voip/skinny(port2000) , / 9.9, wireshark=’skinny') = 

port (20 00); 
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Example 



appid( chat/yahoo', 6.0, chatproc= ’Yahoo’) = 

(('YCHT'c and $yahoo_chat) or first('YCHT'c)) and not 
port(5050); 



appid('chat/icq', 8.5, wireshark='icq ; , chatproc=’ICQ') = 

/^ojicq/c and $icq; 



appidC'chat/tcq 1 , 9.0, wireshark=’icq’, chatproc='ICQ') = 

first('icq’) and not port(25); 

fingerprintfencryption/moujahedeen', 7.0) = 

’begin+gimf+asrar+el+moujahedeen' or 
’begin girnf asrarel rnoujahedeen’; 
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Example 



appid('mail/smtp/to_server r / 8.5, direction = $from server, 
wireshark^smtp') = 

toport(25) and 

( first('helo') or 

first('ehlo') or 

first('data') or 

(IposC'To: 'c) end lpos('From: c)) or 
Ipos('QLUTc) or 
Iposf'mail from :’) or 
lpos( rcpt to: 1 ) ); 
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Example 



$gmail - '<script>D=(top.js&&top.js. in it)?fu notion (d){top.js.P(window'c or 

first('POST /gmail'c) or 
first('GET /gmail'c) or 
'GMAIL_AT=’c or 

/SID=[A-Za-zO-9\-\_]{87> = ;Domain=\.google\. com/c or 

’ G M AI L_ST AT = 1 c or 

'[[V'ctV'c or 

, S=gmail= , c or 

'ain=Vmail. google. com'c or 

'<titSe>Gmairc or 

'GMAIL_RTT='c or 

'GMAIL_LOGIN='c or 

'\nServer: G-E/'c; 

appid('mail/webmail/gmaN', 8.0, we bp roc = 'Gmail') = 

$grnail; 
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# append the mlme_type and HTML title to any of these appids.. 

PARAMS = append = $mime_type, append2=$http_info, 
append3 = $doc_title; 

$web = "web"; 



appid('http/proxy_to„server , / 9.1, $web, direction = $proxy_to_sen/er) 

$webproxy_to_server 

appidChttp/proxyJO-dient', 9.1, $web, direction = $proxy_to_client) = 

$webproxy_to_client; 
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Tvoe Option 



Third parameter is the type; if missing, it takes up to the first slash as 
the type 

appidChttp/response', 9.2, $web) = 

$http and 

not ('x-cache' or 'x-forward' or ’get /' or 
'post /' or 'get http' or 'post http'); 

appid('http/response/partiar, 9.1, $web) = 

$http and $http partial ; 
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Appid utility 



appid options: 

—help 

-“list-all 

— list-appids 

“-list-fingerprints 

-list-types 

-list-levels 

-unit-test 

—quiet 

— appidLfname arg 
-input-file arg 
— datadir arg 



this help message 

list all the application/fingerprint names and 
levels 

list all the application names (no fingerprints) 
list all the application names (no appids) 
list all the application types 
list all the application levels 
perform unit tests with data in the heirachy 
'datadir', with files matching 'filespec' 
don't print any load messages 
location of appid. cfg 
input file to test 

The test data directory. Defaults to 



$(XSCQRE_TEST_DATA_DIR)/appids 
—filespec arg (=.*\.ul24) A regular expression to match against files to 

check 

—noexit arg (=0) do not stop on the first error 
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AoDid Validation 



appid sample. ul24 
Loading appids 

-> Loading : /home/oper/xkeyscore/config/dictionarles/appid/appid_definitions.cfg 
-> Loading : /home/oper/xkeyscore/config/dictionaries/appid/anonymizer. appid 
-> Loading : /home/oper/xkeyscore/config/dictionaries/app5d/bulletin_board. appid 
-> Loading : /home/oper/xkeyscore/conflg/dictionaries/appid/tao_vpn. appid 
-> Loading : /home/oper/xkeyscore/config/dictionaries/appid/tdmoip. appid 
-> Loading : /home/oper/xkeyscore/config/dictionaries/appid/terminal. appid 
-> Loading : /home/oper/xkeyscore/config/dictionaries/appid/voip.appid 
-> Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg 
Finished loading appids 
Filename: sample. ul24 

Appid: encryption/https 

Total Size: 19.36Kbits 
Total Time: O.Olsecs 
Rate: 1.936Mbits/s 
Overall performance: 

Total Time: O.Olsecs 
Total Bits: 0.01936MbitS 
Overall Rate: 1.936M bits/s 
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'cnrry> 

<Lype> !oco_l 
<naTne> eta t/i rc </name> 



:olear: c/Lvoe^- 



<eategory> application ic 
< i r i f c r ma t i o n > 
<application type> chan 
<level> 3,1 </level> 
<action> applicat ion_id 
</inf ormal_ion> 



<sel 


ect> 


KW= 


T <P 03 


KVJ= 


T <P03 


KtJ = 


A 

m 

6 

CQ 


KW- 


T <P03 


KW= 


T <F03 


KW= 


T <ros 


Kft- 


A 

m 

6 

CQ 


KW- 


T <P03 


KW= 


T CFOS 


KW= 


T <F0S 


KW- 


A 

^0 

O 

ul 


KW= 


r <P0S 


KW= 


T ire ' 


KTO= 


’ erro 


</select.> 









appid ( 1 cha „/irc 1 , 8.5, wireshark- T ire T , chalproc 

’ o r i vm 3 q 1 ; 



- T InC T ) - 



appid ( ’ chan/irc 1 , 8.1, wireshar k= T ire T , chatproc= T iAC T ) = 

not (porn (110) and T user T ; and 
( f" rsn ( ’ n i c < ’ ) or fi rst ( 1 i son 1 ) or 

f irsn ( 1 v/hois T ) or firsn ( 1 is ire T ) or 
firs _(' ire ’) or iirsL('join ’) or 
firsn ( 1 aunh ') or first (’crypt ') or 
firsn (’ping ’) or first (’pong ’) or 
firsn ( 'privmsg T } or 
firs „ ( ' no „ice aath') or 
(’ire 1 and ‘privmsg 1 and T not ice 1 ) 
or ' error : yon ’ 

) ; 



<deselect> KW= T ap C0110 1 &KW= ' user ' </deselect> 
</en t ry> 
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